1. Field of the Invention (Technical Field)
The present invention relates to image enhancement of digital images, particularly via processing image data acquired via Electro-Optical (EO) systems.
2. Description of Related Art
EO systems are often used for “remote sensing.” The term “remote sensing” generally refers to the acquisition and measurement of data/information related to one or more properties of a phenomenon, object, or material by a recording device not in physical contact with the object under surveillance. Imaging techniques often involve gathering information by measuring electromagnetic fields, electromagnetic radiation, or acoustic energy using cameras, radiometers, scanners, lasers, radio frequency receivers, radar systems, sonar, thermal devices, seismographs, magnetometers, gravimeters, scintillometers, and like instruments. For example, such data can be acquired and interpreted to remotely sense information about features associated with a target.
Intelligence gathering, particularly within strategic, tactical, or otherwise hostile environments, often relies on technology generally referred to as Enhanced Vision (EV) systems. Through the use of imaging sensors, such as Charge-Coupled Device (CCD) cameras, Forward-Looking Infrared (FLIR), vidicon cameras, Low Light Level cameras, laser illuminated cameras, and the like, targets can be acquired and imagery can be processed and viewed at significantly longer ranges than otherwise possible.
With reference to, for example, FLIR systems, remote sensing can refer to the detecting and measuring of electromagnetic energy, usually thermal or photonic, emanating from distant objects made of various materials. Using FLIR imaging, objects can be identified and categorized by, for example, class, type, substance, or spatial distribution.
To facilitate the acquisition and processing of information from EO systems, sensors can be used on a system's front end to generate raw data for processing. Such sensors can be radar imaging sensors, infrared imaging sensors, electro-optic sensors and the like. In each case, information from which image features can be derived can be used to generate image frames which can then be input to, for example, a display system. Image frames can be integrated with other operational features to form a stable display and to allow for such functions as target identification, acquisition, and tracking to be performed. Such systems can be linked to defense systems to provide guidance input and ordnance control.
In conventional EO systems, the sensors used are limited in their resolution by the fixed spacing between sensor elements. Because of the Nyquist frequency of the sensor as determined by element spacing, image artifacts such as aliasing can be evident in the displayed imagery. A similar type of distortion can arise in, for example, a scene containing edge transitions which are so close together that a sensor cannot accurately resolve them. Resultant distortion can manifest itself as color fringes, in a color camera, around an edge or the like, reducing the ability of a viewer to perceive, for example, letters or object outlines with clarity. Range performance of an EO sensor is also often limited by the Nyquist frequency of the sensor, particularly those containing staring focal-plane arrays. In addition, sensor range can be limited by distortion levels or noise associated with sensor construction.
A conventional method of improving the range performance of an EO system is to improve upon the optics of the system. Such improvements include increasing the focal length of the optics and improving the F/number, i.e., the ratio between the focal length and the aperture size (diameter of a lens), of the system. These types of improvements, however, increase the cost and size of the system, which can lead to a design that is too costly or too large to fit the application.
One technique for addressing the range performance and Nyquist frequency limitations of an EO system is to dither the system, such that the system will sample once, then move the sensor over some sub-pixel amount, and then sample again. Such a technique gives the EO system the appearance that the image is sampled twice as often, and, therefore, the Nyquist frequency of the sensor has effectively doubled. This is often implemented using a dither mechanism such as a Fast Scan Mirror (FSM). However, dither mechanisms, such as a FSM, are usually very expensive and are sensitive to vibrations and alignment.
Accordingly, it is desirable to improve the range performance of EO systems while preserving the integrity of existing EO systems.
The base XR (extended range) technology as disclosed in U.S. patent application Ser. No. 09/841,079, referenced above, is an excellent first step toward image fidelity enhancement; however, this approach is limited in a number of ways. First, the base XR algorithm is limited in practice to a very small sub-image of about 64×64 pixels. Second, the base XR algorithm can only improve those pixels within the image that are moving in unison such that if the XR window is tracking a moving target, all other details outside that target are obscured. Third, if one were to try to utilize the base XR algorithm on a fixed wing aircraft, the change in perspective caused by the aircraft's own motion will degrade the XR algorithm's effectiveness. Finally, for target classification purposes, the base XR algorithm is useful for classification of an already detected target, but one cannot effectively employ it until after one has already located the target, particularly because of the small sub-image limitation.
The approach taken by the present invention solves such problems, though it comes at some hardware expense. In order to offer the best fidelity improvement at each location within the image, each pixel needs to be compensated independently of every other pixel. This requires that a unique motion vector be generated for each and every pixel of the output region.
The base XR algorithm of U.S. patent application Ser. No. 09/841,079 employs a motion vector used to align the current (electronically zoomed) input image with a reference image. This works perfectly as long as all the pixels in the input image experience the same motion relative to the sensor. Of course, all pixels that move differently than the average motion of the group will be blurred by the algorithm or at best case, they will receive no improvement.
The present invention dedicates a separate correlation tracking algorithm to each pixel within the desired output image. Using this technique, each pixel receives individual motion compensation based on matching the local neighborhood between the reference frame and the input image.